{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FdgxBgEf9fC3"
   },
   "source": [
    "<center>\n",
    "    <p style=\"text-align:center\">\n",
    "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
    "        <br>\n",
    "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
    "        |\n",
    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
    "        |\n",
    "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
    "    </p>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jsP3EGfq1sZi"
   },
   "source": [
    "# Using a Benchmark Dataset to Build a Custom LLM as a Judge Evaluator\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZJmqp2Po9mQM"
   },
   "source": [
    "In this tutorial, you’ll learn how to build a custom LLM-as-a-Judge Evaluator tailored to your specific use case. While Phoenix provides several [pre-built evaluators](https://arize.com/docs/phoenix/evaluation/how-to-evals/running-pre-tested-evals) that have been tested against benchmark datasets, these may not always cover the nuances of your application.\n",
    "\n",
    "So how can you achieve the same level of rigor when your use case falls outside the scope of standard evaluators?\n",
    "\n",
    "We’ll walk through how to create your own benchmark dataset using a small set of annotated examples. This dataset will allow you to build and refine a custom evaluator by revealing failure cases and guiding iteration.\n",
    "\n",
    "To follow along, you’ll need:\n",
    "\n",
    "- A free [Phoenix Cloud](https://app.arize.com/auth/phoenix/login) account\n",
    "- An OpenAI API Key\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CzYQTdBL3f3O"
   },
   "outputs": [],
   "source": [
    "!pip install -qqqq arize-phoenix arize-phoenix-otel openinference-instrumentation-openai openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1J1Rqzg53x6H"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if not (phoenix_endpoint := os.getenv(\"PHOENIX_COLLECTOR_ENDPOINT\")):\n",
    "    os.environ[\"PHOENIX_COLLECTOR_ENDPOINT\"] = getpass(\"🔑 Enter your Phoenix Collector Endpoint: \")\n",
    "\n",
    "if not (phoenix_api_key := os.getenv(\"PHOENIX_API_KEY\")):\n",
    "    os.environ[\"PHOENIX_API_KEY\"] = getpass(\"🔑 Enter your Phoenix API key: \")\n",
    "\n",
    "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass(\"🔑 Enter your OpenAI API key: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VWVp1oKW30Yg"
   },
   "outputs": [],
   "source": [
    "from phoenix.otel import register\n",
    "\n",
    "# configure the Phoenix tracer\n",
    "tracer_provider = register(\n",
    "    project_name=\"receipt-classification\",\n",
    "    auto_instrument=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gWSWwjCkmdS"
   },
   "source": [
    "# Generate Image Classification Traces\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UdYjpCRIlO1n"
   },
   "source": [
    "In this tutorial, we’ll ask an LLM to generate expense reports from receipt images provided as public URLs. Running the cells below will generate traces, which you can explore directly in Phoenix for annotation. We’ll use GPT-4.1, which supports image inputs.\n",
    "\n",
    "Dataset Information:\n",
    "Jakob (2024). Receipt or Invoice Dataset. Roboflow Universe. CC BY 4.0. Available at: https://universe.roboflow.com/jakob-awn1e/receipt-or-invoice (accessed on 2025‑07‑29)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UsbFytJLvl32"
   },
   "outputs": [],
   "source": [
    "urls = [\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/8M5px2yLoNtZ6gOQ2r1D/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/8EVgYMNObyV6kLqBNeFG/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/86aohWmcEfO0XkflO8AB/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/1eGPBChz7wvovQROk2l8/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/0WqR2GSfGmxWB7ozo3Pj/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/FAEJRtviIboCYSKFZcEZ/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/0AoEaFy8FAw6DVieWCa8/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/0Q3hAyNwXNpHTeoWU7fz/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/2r876u4WpaCYFdMPwieK/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/2ZWeE0yO0oJUDtpgEAPY/original.jpg\",\n",
    "    \"https://source.roboflow.com/HahhKcbQqdf8YAudM4kU3PuVCS72/37PF6xfHyuqzIBdO7Kgw/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/6mo4M0nJeKZEsdKrRfsR/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/5ezJ8tUBGbNnt0jZi2JU/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/4BCIWGazhCj03oTMWboO/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/4B8vXJNwJ7ZuHEWyjgAv/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/2EpeKbAqsSwciH2IHGyV/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/2LP3g9rKZrYDkNB3I78c/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/1hT6iLEIAFBw8W70u2FY/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/1zaKpaDhRPxkiIDTvMuc/original.jpg\",\n",
    "    \"https://source.roboflow.com/Zf1kEIcRTrhHBZ7wgJleS4E92P23/1hF1R2Pt41hnlqhlXLDD/original.jpg\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "id": "daEneqTU1T1q"
   },
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI()\n",
    "\n",
    "\n",
    "def extract_receipt_data(input):\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"gpt-4.1\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\n",
    "                        \"type\": \"text\",\n",
    "                        \"text\": \"Analyze this receipt and return a brief summary for an expense report. Only include category of expense, total cost, and summary of items\",\n",
    "                    },\n",
    "                    {\n",
    "                        \"type\": \"image_url\",\n",
    "                        \"image_url\": {\n",
    "                            \"url\": input,\n",
    "                        },\n",
    "                    },\n",
    "                ],\n",
    "            }\n",
    "        ],\n",
    "        max_tokens=500,\n",
    "    )\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jk1Skh1CFafZ"
   },
   "outputs": [],
   "source": [
    "for url in urls:\n",
    "    extract_receipt_data(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oUoJBZgleEe"
   },
   "source": [
    "# Create Benchmarked Dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AemZrhRh_SpU"
   },
   "source": [
    "After generating traces, open Phoenix to begin annotating your dataset. In this example, we’ll annotate based on \"accuracy\", but you can choose any evaluation criterion that fits your use case. Just be sure to update the query below to match the annotation key you’re using—this ensures the annotated examples are included in your benchmark dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z3Abt5fGDPxk"
   },
   "source": [
    "![Annotated Data](https://storage.googleapis.com/arize-phoenix-assets/assets/images/build-your-first-eval-tutorial-1.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AkVsgLquXWi8"
   },
   "outputs": [],
   "source": [
    "from phoenix.client import Client\n",
    "from phoenix.client.types import spans\n",
    "\n",
    "px_client = Client()\n",
    "# replace \"accuracy\" if you chose to annotate on different criteria\n",
    "query = spans.SpanQuery().where(\"annotations['accuracy']\")\n",
    "spans_df = px_client.spans.get_spans_dataframe(\n",
    "    query=query, project_identifier=\"receipt-classification\"\n",
    ")\n",
    "annotations_df = px_client.spans.get_span_annotations_dataframe(\n",
    "    spans_dataframe=spans_df, project_identifier=\"receipt-classification\"\n",
    ")\n",
    "full_df = annotations_df.join(spans_df, how=\"inner\")\n",
    "\n",
    "dataset = px_client.datasets.create_dataset(\n",
    "    dataframe=full_df,\n",
    "    name=\"annotated-receipts\",\n",
    "    input_keys=[\"attributes.input.value\"],\n",
    "    output_keys=[\"attributes.llm.output_messages\"],\n",
    "    metadata_keys=[\"result.label\", \"result.score\", \"result.explanation\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7FzMJND2lnHY"
   },
   "source": [
    "# Create evaluation template\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "38Sm4UBeAJrd"
   },
   "source": [
    "Next, we’ll create a baseline evaluation template and define both the task and the evaluation function. Once these are set up, we’ll run an experiment to compare the evaluator’s performance against our ground truth annotations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-uIfbY7DFinO"
   },
   "outputs": [],
   "source": [
    "from phoenix.evals.templates import (\n",
    "    ClassificationTemplate,\n",
    "    PromptPartContentType,\n",
    "    PromptPartTemplate,\n",
    ")\n",
    "\n",
    "rails = [\"accurate\", \"almost accurate\", \"inaccurate\"]\n",
    "classification_template = ClassificationTemplate(\n",
    "    rails=rails,  # Specify the valid output labels\n",
    "    template=[\n",
    "        # Prompt part 1: Task description\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.TEXT,\n",
    "            template=\"\"\" You are an evaluator tasked with assessing the quality of a model-generated expense report based on a receipt.\n",
    "Below is the model’s generated expense report and the input image:\n",
    "---\n",
    "MODEL OUTPUT (Expense Report): {output}\n",
    "\n",
    "---\n",
    "INPUT RECEIPT: \"\"\",\n",
    "        ),\n",
    "        # Prompt part 2: Insert the image data\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.IMAGE,\n",
    "            template=\"{image}\",  # Placeholder for the image URL\n",
    "        ),\n",
    "        # Prompt part 3: Define the response format\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.TEXT,\n",
    "            template=\"\"\" Evaluate the following three aspects and assign one of the following labels for each. Only include the label:\n",
    "- **\"accurate\"** – Fully correct\n",
    "- **\"almost accurate\"** – Mostly correct\n",
    "- **\"inaccurate\"** – Substantially wrong\n",
    "            \"\"\",\n",
    "        ),\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(str(classification_template))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VL6oXupczt-O"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from phoenix.evals import (\n",
    "    LLM,\n",
    "    ClassificationEvaluator,\n",
    "    create_evaluator,\n",
    ")\n",
    "\n",
    "llm = LLM(provider=\"openai\", model=\"gpt-4o\")\n",
    "\n",
    "\n",
    "def run_accuracy_eval(input, example):\n",
    "    input_data = json.loads(example[\"input\"][\"attributes.input.value\"])\n",
    "    output_data = example[\"output\"][\"attributes.llm.output_messages\"]\n",
    "\n",
    "    # image_url = input_data[\"messages\"][0][\"content\"][1][\"image_url\"][\"url\"]\n",
    "\n",
    "    row = {\n",
    "        \"image\": input_data[\"messages\"][0][\"content\"][1][\"image_url\"][\"url\"],\n",
    "        \"output\": output_data,\n",
    "    }\n",
    "\n",
    "    accuracy = ClassificationEvaluator(\n",
    "        name=\"accuracy\",\n",
    "        llm=llm,\n",
    "        prompt_template=str(classification_template),\n",
    "        choices={\"accurate\": 1.0, \"almost accurate\": 0.5, \"inaccurate\": 0},\n",
    "    )\n",
    "\n",
    "    result = accuracy.evaluate(row)\n",
    "    return result[0]\n",
    "\n",
    "\n",
    "@create_evaluator(name=\"close_match\")\n",
    "def close_match(output, metadata):\n",
    "    expected_label = metadata[\"result.label\"]\n",
    "    predicted_label = output[\"label\"]\n",
    "    return 1 if expected_label == predicted_label else 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vCybZkrckjd6"
   },
   "outputs": [],
   "source": [
    "from phoenix.client import AsyncClient\n",
    "\n",
    "async_client = AsyncClient()\n",
    "\n",
    "experiment = await async_client.experiments.run_experiment(\n",
    "    dataset=dataset,\n",
    "    task=run_accuracy_eval,\n",
    "    experiment_name=\"baseline\",\n",
    "    evaluators=[close_match],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZASPVzDLQiQE"
   },
   "source": [
    "#Iteration 1 to improve evaluator prompt template\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rXTk3celAZHW"
   },
   "source": [
    "Next, we’ll refine our evaluation prompt template by adding more specific instructions to classification rules. We can add these rules based on gaps we saw in the previous iteration. This additional guidance helps improve accuracy and ensures the evaluator's judgments better align with human expectations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aN-zl8FzjhPf"
   },
   "outputs": [],
   "source": [
    "classification_template = ClassificationTemplate(\n",
    "    rails=rails,  # Specify the valid output labels\n",
    "    template=[\n",
    "        # Prompt part 1: Task description\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.TEXT,\n",
    "            template=\"\"\" You are an evaluator tasked with assessing the quality of a model-generated expense report based on a receipt.\n",
    "Below is the model’s generated expense report and the input image:\n",
    "---\n",
    "MODEL OUTPUT (Expense Report): {output}\n",
    "\n",
    "---\n",
    "INPUT RECEIPT: \"\"\",\n",
    "        ),\n",
    "        # Prompt part 2: Insert the audio data\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.IMAGE,\n",
    "            template=\"{image}\",  # Placeholder for the image URL\n",
    "        ),\n",
    "        # Prompt part 3: Define the response format\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.TEXT,\n",
    "            template=\"\"\" Evaluate the following and assign one of the following labels for each. Only include the label:\n",
    "- **\"accurate\"** – Total price, itemized list, and expense category are all accurate. All three must be correct to get this label.\n",
    "- **\"almost accurate\"** – Mostly correct but with small issues. For example, expense category is too vague.\n",
    "- **\"inaccurate\"** – Substantially wrong or missing information. For example, incorrect total price.\n",
    "            \"\"\",\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yzewNOiJSYIe"
   },
   "outputs": [],
   "source": [
    "experiment = await async_client.experiments.run_experiment(\n",
    "    dataset=dataset,\n",
    "    task=run_accuracy_eval,\n",
    "    experiment_name=\"improved template\",\n",
    "    evaluators=[close_match],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OUlH0oT3SnLq"
   },
   "source": [
    "#Iteration 2 to improve evaluator prompt template\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EtCVayyRA1j8"
   },
   "source": [
    "To further improve our evaluator, we’ll introduce few-shot examples into the evaluation prompt. These examples help highlight common failure cases and guide the evaluator toward more consistent and generalized judgments.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L_KB0S_lSZfP"
   },
   "outputs": [],
   "source": [
    "classification_template = ClassificationTemplate(\n",
    "    rails=rails,  # Specify the valid output labels\n",
    "    template=[\n",
    "        # Prompt part 1: Task description\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.TEXT,\n",
    "            template=\"\"\" You are an evaluator tasked with assessing the quality of a model-generated expense report based on a receipt.\n",
    "Below is the model’s generated expense report and the input image:\n",
    "---\n",
    "MODEL OUTPUT (Expense Report): {output}\n",
    "\n",
    "---\n",
    "INPUT RECEIPT: \"\"\",\n",
    "        ),\n",
    "        # Prompt part 2: Insert the audio data\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.IMAGE,\n",
    "            template=\"{image}\",  # Placeholder for the image URL\n",
    "        ),\n",
    "        # Prompt part 3: Define the response format\n",
    "        PromptPartTemplate(\n",
    "            content_type=PromptPartContentType.TEXT,\n",
    "            template=\"\"\" Evaluate the following three aspects and assign one of the following labels for each. Only include the label:\n",
    "- **\"accurate\"** – Total price, itemized list, and expense category are accurate. All three must be correct to get this label.\n",
    "  An incorrect category is one that is overly vague (e.g., “Miscellaneous”, \"Supplies\") or does not accurately reflect the itemized list.\n",
    "  For example, \"Dining and Entertainment\" should not be grouped together if the itemized list only includes food.\n",
    "  Reasonable general categories like “Office Supplies” or “Groceries” are acceptable if they align with the listed items.\n",
    "\n",
    "- **\"almost accurate\"** – Mostly correct but with small issues. For example, expense category is too vague.\n",
    "  If a category includes extra fields (ex: \"Dining and Entertainment\", but the receipt only includes food) mark this as almost correct.\n",
    "- **\"inaccurate\"** – Substantially wrong or missing. For example, incorrect total price or one of more of the items is missing makes the total result inaccurate.\n",
    "            \"\"\",\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hqKXWt-VzR_R"
   },
   "outputs": [],
   "source": [
    "experiment = await async_client.experiments.run_experiment(\n",
    "    dataset=dataset,\n",
    "    task=run_accuracy_eval,\n",
    "    experiment_name=\"few shot improved template\",\n",
    "    evaluators=[close_match],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WD5A8ILUBGSN"
   },
   "source": [
    "# Final Results\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3dxEKV2YBH7V"
   },
   "source": [
    "Once your evaluator reaches a performance level you're satisfied with, it's ready for use. The target score will depend on your benchmark dataset and specific use case. That said, you can continue applying the techniques from this tutorial to refine and iterate until the evaluator meets your desired level of quality.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UQrp-RXBDUW7"
   },
   "source": [
    "![Final Results](https://storage.googleapis.com/arize-phoenix-assets/assets/images/build-your-first-eval-tutorial-2.png)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
